Papers with language modeling datasets
Plug and Play Knowledge Distillation for kNN-LM with External Logits (2022.aacl-short)
Copied to clipboard
| Challenge: | Despite the promising evaluation results by knowledge distillation (KD) in natural language understanding (NLU) and sequence-to-sequence (seq2sequ) tasks, KD for causal language modeling (LM) remains a challenge. |
| Approach: | They propose to use external logits to improve a student's kNN-LM by leveraging teacher's knowledge at test time. |
| Outcome: | The proposed method improves a student's kNN-LM in multiple language modeling datasets and improves perplexity. |
Scaling Hidden Markov Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Hidden Markov models are a fundamental tool for sequence modeling that separates the hidden state from the emission structure. |
| Approach: | They propose methods for scaling hidden Markov models to massive state spaces while maintaining efficient exact inference and effective regularization. |
| Outcome: | The proposed methods are much more accurate than previous HMMs and n-gram-based methods, making progress towards the performance of state-of-the-art NN models. |